adjustments for building DB and using newer version of irsx #24
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fgregg
commented
Jul 17, 2019
| @@ -0,0 +1,31 @@ | |||
| export IRSX_CACHE_DIRECTORY=. | |||
Author
There was a problem hiding this comment.
This is a Makefile I made for building the whole db.
fgregg
commented
Jul 17, 2019
| perl -i.bak -p -e 's/SILVERCREST ASSET ,AMAGEMENT/SILVERCREST ASSET MANAGEMENT/g' $@ | ||
| python manage.py enter_yearly_submissions 2014 | ||
|
|
||
| download_filings_% : |
Author
There was a problem hiding this comment.
It's a lot faster to download the files using this the aws-cli and then run 'load_filings' then have load filings be responsible also for downloading.
On the downside, this requires folks to have an AWS account.
fgregg
commented
Jul 17, 2019
| # }, | ||
| # }, | ||
| # ] | ||
| TEMPLATES = [ |
Author
There was a problem hiding this comment.
This needed to be uncommented in order for any of the management commands to run.
fgregg
commented
Jul 17, 2019
| # row['canonical_version'] = CANONICAL_VERSION | ||
| #else: | ||
| # row['is_canonical'] = False | ||
| row['versions'] = row['versions'].split(';') |
Author
There was a problem hiding this comment.
this is for using the github version of irsx.
fgregg
commented
Jul 17, 2019
| @@ -1,4 +1,5 @@ | |||
| from django.db import models | |||
| from django.contrib.postgres.fields import ArrayField | |||
Author
There was a problem hiding this comment.
i imagine you may not want to tie yourself to postgres. there is a portable version of jsonfield that could be used here.
fgregg
commented
Jul 17, 2019
| try: | ||
| FILE_SYSTEM_BASE = settings.FILE_SYSTEM_BASE | ||
| except ImportError: | ||
| except AttributeError: |
Author
There was a problem hiding this comment.
this is the correct exception catcher.
fgregg
commented
Jul 17, 2019
| partname = part | ||
| partdata = sked['schedule_parts'][part] | ||
| #print("part %s %s" % (partname, partdata)) | ||
| #self.stdout.write("part %s %s" % (partname, partdata)) |
Author
There was a problem hiding this comment.
just using the django command guidance here.
fgregg
commented
Jul 17, 2019
| self.stdout.write("run_filing %s" % object_id) | ||
|
|
||
| # if we get a bad xml file, delete the file and retry once | ||
| try: |
Owner
|
Thanks so much for sharing this @forest Gregg <[email protected]>, excited
to take a look. in the next few days. You're reading is correct; at the
outset I didn't want to require an amazon account, but the download with
aws is so much better with one it probably makes sense to use it (and
everyone working with this data at scale probably has one anyways).
…On Wed, Jul 17, 2019 at 3:02 PM Forest Gregg ***@***.***> wrote:
Hi @jsfenfen <https://github.com/jsfenfen>, this PR has a number of
different pieces that you may or not be interested in. I can split things
apart to just get you what you do care about.
------------------------------
You can view, comment on, or merge this pull request online at:
#24
Commit Summary
- adjustments for building DB and using newer version of irsx
File Changes
- *M* README.md
<https://github.com/jsfenfen/990-xml-database/pull/24/files#diff-0>
(2)
- *A* irsdb/Makefile
<https://github.com/jsfenfen/990-xml-database/pull/24/files#diff-1>
(31)
- *M* irsdb/irsdb/settings.py
<https://github.com/jsfenfen/990-xml-database/pull/24/files#diff-2>
(30)
- *M* irsdb/metadata/management/commands/load_metadata.py
<https://github.com/jsfenfen/990-xml-database/pull/24/files#diff-3>
(6)
- *M* irsdb/metadata/models.py
<https://github.com/jsfenfen/990-xml-database/pull/24/files#diff-4>
(16)
- *M* irsdb/metadata/views.py
<https://github.com/jsfenfen/990-xml-database/pull/24/files#diff-5>
(4)
- *M* irsdb/return/management/commands/load_filings.py
<https://github.com/jsfenfen/990-xml-database/pull/24/files#diff-6>
(38)
Patch Links:
- https://github.com/jsfenfen/990-xml-database/pull/24.patch
- https://github.com/jsfenfen/990-xml-database/pull/24.diff
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24?email_source=notifications&email_token=AAE4AJFEEWAZBBZE6NS7F63P76JFVA5CNFSM4IEU535KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G73QODA>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAE4AJELCDXJB5JFF7BU4PTP76JFVANCNFSM4IEU535A>
.
|
|
Would recommend splitting off the change from |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @jsfenfen, this PR has a number of different pieces that you may or not be interested in. I can split things apart to just get you what you do care about.